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I.  RESEARCH  ACCOMPLISHMENTS 

Research  for  ONR  Contract  "Statistical  Inference  From  Sampled  Data"  has  been  concerned  in  the 
past  6  years  (1984-1989)  with  studies  in  the  general  area  of  statistical  signal  processing.  Contributions 
were  made  in  a  wide  range  of  topics  many  of  which  were  motivated  by  practical  problems  in 
communication  systems  and  digital  signal  processing.  The  following  is  a  list  of  the  main  topics  on  which 
the  research  was  focused: 

A.  Frequency  Wa^emumbCx  S{V''tral  Estimation  Using  Random  Acoustic  Arrays. 

B.  Spread  Spectrum  Communication  Systems. 

C.  Probability  Density  and  Regression  Estimation  for  Dependent  Data. 

D.  Parametric  and  Nonparametrc  Spectral  Estimation. 

E.  Adaptive  Linear  Estimation  Algorithms  for  Dependent  Processes 

F.  Inference  for  Continuous-Time  Processes  from  Sampled  Data. 


The  research  under  this  contract  resulted  in  the  publication  of  19  papers  in  mathematical,  statistical, 
and  engineering  journals.  In  addition,  several  papers  were  presented  and  subsequently  published  in 
proceedings  of  conferences.  Copies  of  these  works  were  routinely  sent  to  the  Office  of  Naval  Research, 
Mathematical  Sciences  Division.  The  following  is  a  list  of  the  journal  publications  under  this  contract. 
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1.  E.  Masry,  "The  Estimation  of  the  Frequency- Wavenumber  Spectrum  Using  Random  Acoustic 
Arrays  -  Part  I:  Performance  of  Beampower  Pattern  Estimators,"  J.  Acoustical  Society  of  America, 
Vol.  76  (July  1984),  pp.  139-149. 


2.  E.  Masry,  "Closed-Form  Analytical  Results  for  the  Rejection  of  Narrow-Band  Interference  in  PN 
Spread-Spectrum  Systems  -  Part  I:  Linear  Prediction  Filters,"  IEEE  Trans.  Communications, 
COM-32  (August  1984),  pp.  888-896. 


3.  E.  Masry,  "The  Estimation  of  the  Frequency- Wavenumber  Spectrum  Using  Random  Acoustic 
Arrays  -  Part  II:  A  Class  of  Consistent  Estimators,"  J.  Acoustical  Society  of  America,  Vol.  76 
(October  1984),  pp.  1123-1131. 


4.  E.  Masry,  "A  Class  of  Generalized  Szasz  Operators  and  their  Convergence  Properties," 
Mathematica,  Vol.  13  (1984),  pp.  45-56. 


5.  E.  Masry,  "Closed-Form  Analytical  Results  for  the  Rejection  of  Narrow-Band  Interference  in  PN 
Spread-Spectrum  Systems  -  Part  II:  Linear  Interpolation  Filters,"  IEEE  Trans.  Communications, 
COM-33  (January  1985),  pp.  10-19. 


6.  D.F.  Gingras  and  E.  Masry,  "Spectral  Density  Estimation  from  Nonlinearly  Observed  Data,"  J.  Time 
Series  Analysis,  Vol.  6  (1985),  pp.  63-80. 


7.  E.  Masry  and  L.  Milstein,  "Performance  of  DS  Spread-Spectrum  Receiver  Emi  >vtr,i  Interference- 
Suppression  Filters  under  a  Worst-Case  Jamming  Condition,"  IEEE  Trans.  imunications, 
COM-34(January  1986),  pp.  13-21. 


8.  E.  Masry,  "Recursive  Probability  Dens^y  Estimation  for  Weakly  Dependent  Stationary  Processes," 
IEEE  Trans.  Information  Theory,  IT-32(March  1986),  pp. 


9.  E.  Masiy  and  R.  Rao,  "On  the  Choice  of  the  Timeout  Distribution  in  Decentralized  Random  Access 
Systems,"  IEEE  Trans.  Communications,  COM-34(May  1986),  pp.501-504. 

10.  E.  Masry  and  B.  Picinbono,  "Linear/Nonlinear  Forms  and  the  Normal  Law:  Characterization  by 
High  Order  Correlation:" ,"  *nnak  institute  Statist  Math..  Vo]  39(1987),  Part  A,  pp.  417  42S. 

11.  E.  Masry  and  L.  Gyorfi,  "Strong  Consistency  and  Rates  for  Recursive  Probability  Density 
Estimators  of  Stationary  Processes,"  J.  Multivariate  Analysis,  Vol.  22(June  1987),  pp.  79-93. 
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12.  E.  Masry,  "Almost  Sure  Convergence  of  Recursive  Density  Estimators  for  Stationary  Mixing 
Processes,  "  Statistics  &.  Probability  Letters,  Vol.  5(1987),  pp.  249-254. 

13.  D.F.  Gingras  and  E.  Masry,  "Autoregressive  Spectral  Estimation  in  Additive  Noise,"  IEEE  Trans. 
Acoustics,  Speech,  and  Signal  Processing,  ASSP-36(April  1988),  pp.  490-501. 

14.  S.  Cambanis  and  E.  Masry,  "The  Performance  of  Discrete-Time  Predictors  of  Continuous-Time 
Stauonary  Processes,”  IEEE  Trans.  Information  Theory,  IT-34(July  1988),  pp.  655-668. 

15.  A.  Krieger  and  E.  Masry,  "Convergence  Analysis  ot  Adaptive  Linear  Estimation  for  Dependent 
Stationary  Processes,"  IEEE  Trans.  Information  Theory,  IT-34(July  1988),  pp.  642-654. 

16.  E.  Masry,  "Random  Sampling  of  Continuous-Parameter  Stationary  Processes:  Statistical  Properties 
of  Joint  Density  Estimators,"  J.  Multivariate  Analysis,  Vol.  26(August  1988),  pp.  133-165. 

17.  E.  Masry,  "Nonparametric  Estimation  of  Conditional  Probability  Densities  and  Expectations  of 
Stationary  Processes:  Strong  Consistency  and  Rates,"  J.  Stochastic  Processes  &  Applies.,  Vol. 
32(1989),  pp.  109-127. 

18  A.  Krieger  and  E.  Masry,  "Constrained  Adaptive  Filtering  Algorithms:  Asymptotic  Convergence 
Properties  for  Dependent  Data,"  IEEE  Trans.  Information  Theory,  lT-35(Novembcr  1989),  pp. 
1166-1176. 

19.  E.  Masry  and  S.  Cambanis,  "Trapezoidal  Monte  Carlo  Integration,"  SIAM  J.  Numerical  Analysis, 
Vol.  27(February  1990). 
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II.  DESCRIPTIVE  SUMMARY  OF  PROBLEMS  AND  RESULTS 

A  descriptive  summary  of  the  research  problems  studied  under  this  contract  and  the  nature  of  the 
results  obtained  is  now  presented  for  each  of  the  principal  areas  listed  in  Section  I. 

A.  Frequency- Wavenumber  Spectral  Estimation  Using  Random  Acoustic  Arrays 

The  use  of  an  array  of  sensors  for  underwater  acoustic  measurement  is  standard.  In  an  array  ot 
equally-spaced  elements,  the  use  of  spacing  greater  than  half-wavelength  leads  to  wavenumber  aliasing 
[1  ];  aliasing  can  be  reduced  by  using  aperiodic  arrays.  There  has  been  increasing  interest  in  random  arrays 
[2][3]  which  are  aperiodic  arrays  whose  elements’  positions  are  selected  at  random.  The  interest  is 
particularly  motivated  by  the  idea  that  a  field  of  randomly  deployed,  freely  drifting,  sonobuoys  ca  be  used 
to  form  a  high  gain  array  for  underwater  acoustic  measurements.  An  array  is  totally  random  if  the 
sensors’  locations  are  realizations  of  independent  identically  distributed  random  vectors  [2],  Thom  et  al 
[2]  derived  fundamental  properties  of  random  arrays  such  as  the  mean  and  variance  of  the  pattern 
function  and  its  power. 

Among  the  generic  signal  processing  tasks  of  an  array  system  is  the  estimation  of  the  frequency- 
wavenumber  spectrum  of  the  ambient  noise  field.  When  the  array  system  is  random,  the  estimation  of  the 
frequency-wavenumber  spectrum  from  data  collected  at  its  output  is  quite  complex.  Hinich  [3]  proposed  a 
conventional  (periodogram)  approach  and  showed  that  the  estimator  is  asymptotically  unbiased  provided 
that  the  density  of  the  number  of  sensors  per  unit  area  tends  to  infinity;  thus  the  sensors  must  cover  the 
plane  densely.  The  practical  significance  of  this  approach  is  clearly  doubtful. 

In  [4][5]  we  provided  a  quadratic-mean  convergence  analysis  of  appropriate  estimates  of  the 
frequency-wavenumber  spectrum  of  the  ambient  noise  field,  in  n-dimensional  Euclidean  space,  from  data 
collected  by  a  totally  random  n-dimensional  array.  In  contrast  to  [3]  the  density  of  the  number  of  sensors 
per  unit  volume  was  allowed  to  be  arbitrary.  For  a  class  of  frequency-wavenumber  spectral  estimates,  we 
provided  closed-form  expression  for  the  asymptotic  bias  and  covariance  of  these  estimates.  The  estimates 
were  shown  to  be  consistent  in  quadratic-mean  as  the  observation  interval  length  T  and  the  number  of 
sensors  M  tend  to  infinity  ( the  array  can  be  arbitrarily  sparse).  The  rate  of  quadratic  mean  convergence 
was  determined.  The  dependence  of  the  mean-square  estimation  error  on  the  parameters  of  the  system, 
such  as  the  observation  length  T,  the  number  of  sensors  M,  the  probability  density  of  the  sensors’ 
positions,  and  the  number  of  sensors  per  unit  volume  was  investigated. 
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B.  Spread  Spectrum  Communication  Systems 

Spread  Spectrum  communication  systems  offer  immunity  against  narrow-band  interference.  This 
can  be  further  improved  by  employing  linear  mean-square  estimation  techniques [6]-[ 8]  for  the  estimation 
ooh  subsequent  suppression  of  the  interference.  These  techniques  exploit  the  distinct  spectral 
characteristics  of  the  interference,  which  is  generally  narrow-band,  and  that  of  *he  pseudo-noise  (PN)  or 
direct  sequence  (DS)  signals  which  are  broad-band.  When  the  interference  is  modeled  as  a  multiple  tone, 
the  improvement  in  the  performance  of  the  system  due  to  the  employment  of  linear  interference- 
suppression  filters  has  been  thoroughly  studied  in  [6]-[8]. 

In  [9][  10]  we  provided  an  analytical  study  of  the  performance  of  PN  spread  spectrum 
communication  systems,  using  interference-rejection  filters,  when  the  interference  is  modeled  as  a 
stationary  process  with  arbitrary  spectral  density.  In  [9]  linear  predictive  filters  were  employed  and  in 
[10]  linear  interpolative  filters  were  used.  Specialized  results  were  obtained  for  interferences  which  are 
either  bandlimited  or  have  rational  spectral  densities.  In  each  case,  the  additional  improvement  in  the 
performance  of  the  receiver  was  investigated  particularly  its  dependence  on  such  factors  as  the 
interference  power  and  the  number  of  taps  of  the  filter. 

In  [11]  we  investigated  the  spectral  characteristics  of  the  worst-case  jammer,  assumed  to  be  a 
narrow-band  Gaussian  process,  under  a  joint  power  and  bandwidth  constraints.  An  expression  for  the 
average  probability  of  error  is  derived  for  a  DS  spread-spectrum  communication  system.  The 
effectiveness  of  the  suppression  filter  under  these  conditions  was  studied  and  illustrated  for  various 
combinations  of  system  parameters. 

C.  Probability  Density  and  Regression  Estimation  for  Dependent  Observations 

Probability  density  and  regression  estimation  play  an  important  role  in  communication  theory, 
pattern  recognition  and  classification  [12]-[18]. 

Nonparametric  density  estimation  for  stationary  random  processes  (dependent  observations)  has 
been  receiving  increasing  attention  in  recent  years.  Rosenblatt  [19]  considered  nonrecursive  kernel-type 
density  estimation  for  Markov  processes;  Takahata  [20],  Robinson  [21][22],  and  Castellana  and 
Lcadbetter  [23]  dealt  with  similar  estimators  for  mixing  processes.  These  works  are  primarily  concerned 
with  establishing  quadratic-mean  convergence  and  asymptotic  normality  of  nonrecursive  density 
estimators. 

Under  the  current  contract  we  initiated  an  investigation  into  the  the  statistical  properties  of  recursive 
kernel  density  estimators  for  stationary  processes.  In  [24]  we  established  the  quadratic-mean 
convergence,  along  with  the  asymptotic  expressions  for  the  bias  and  covariance  of  such  estimators,  for 
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mixing  processes.  Also,  asymptotic  normality  was  established.  In  [25]  [26]  we  derived  sharp  almost  sure 
convergence  rates  for  the  marginal  density  estimators  of  vector-valued  stationary  mixing  processes.  In 
[27]  we  investigated  the  almost  sure  convergence  of  recursive  kernel-type  estimators  for  the  joint 
probability  densities,  conditional  densities,  and  conditional  expectations  of  scalar-valued  stationary 
mixing  processes.  Specifically,  For  each  integer  m  >  1  and  integers  0  =  /  ]  <i2<  <  i„  let 

. xm;  i{  .....  im)  be  the  joint  probability  density  function  of  the  random  variables 

A ’ii  , . . . ,  Xim,  which  is  assumed  to  exist.  For  any  integer  p,  \  <p  <  m,  put  if  =  {i\ . ip )  and 

=  (ip+\  , . . . ,  rm).  The  conditional  probability  density  function  of  XJ  =  (Xj+i  . XJ+im)  given 

XJ  =  (Xj+i^  , ... ,  Xj+i)  is  denoted  by 


f(x2\xi)  4 /(£2;im-Pl£i;p 


f(x\i_m) 
fix  r,  p 


where  £i  e  Rp,  x2  e  Rm  p  and  x  =  (_x\,  x2)  e  Rm.  Let  q  be  a  Borel  measurable  function  on  Rm  p  such 
that  E  |  q(XJ)  I  <  °°.  The  conditional  expectation  of  q(XJ)  given  XJ  =  u  is  denoted  by 

Q(u)  =  E[q(Xj)\X'  =  u]. 

A  special  case  of  interest  is  when  m  =p+ 1  and  qiy)=y  for  which  Q(u)  =  E[Xj+i  \  Xj  =  u]  is  the  usual 
regression  function.  In  [27]  recursive  kernel  estimators  fnix\  i_m),  fnixi  |  ^i),  and  Qn(u)  are  defined  and 
their  strong  consistency  and  almost  sure  convergence  rates  are  established. 


D.  Parametric  and  Nonparametric  Spectral  Estimation 

In  the  area  of  nonparametric  spectral  estimation,  we  considered  in  [28]  the  problem  of  estimating 
the  spectral  density  function  0(A)  of  a  stationary  Gaussian  process  from  observations  that  have 

been  distorted  by  a  known  zero-memory  nonlinearity.  Such  distortions  may  be  due  to  the  presence  of  a 
nonlinear  device  in  the  data  communication  system;  an  n-bit  quantizer  is  an  example.  Certain  aspects  of 
this  problem  were  considered  earlier  by  Rodemich  [29],  McNeil  [30]  and  Brillinger  [31].  In  [28]  we 
considered  the  case  where  the  nonlinearity /is  real,  bounded,  odd,  and  nondecreasing  function  on  the  real 
line.  The  output  process  is  [Yk=f  (X*)}r=-~  and  the  problem  is  the  estimation  of  the  spectral  density  0(A) 
of  the  input  process  X  from  the  knowledge  of  /  and  a  finite  set  of  observations  [TiJ^i  of  the  output 
process  Y.  In  [28]  we  introduced  a  class  of  spectral  estimates  and  its  quadratic-mean  consistency  is 
established  along  with  the  asymptotic  expressions  for  the  bias  and  covariance.  In  addition,  the  asymptotic 
normality  of  the  spectral  estimates  is  derived.  Our  results  extend  those  of  Brillinger  [31],  where  a  signum 
nonlinearity  /  (x)=sign[j:  ]  was  considered,  and  refine  and  correct  those  of  [29]  [30]. 


In  the  area  of  parametric  spectral  estimation  we  considered  in  [32]  the  problem  of  estimating  the 
spectral  density  of  an  autoregressive  (AR)  stationary  process  from  a  finite  set  of  noisy  observations.  A 
modified  spectral  estimator  based  on  the  high-order  Yule-Walker  equations  was  considered.  Joint 
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asymptotic  normality  of  the  spectral  estimator  is  established  along  with  -  precise  expression  for  its 
covariance  matrix.  The  work  extends  the  results  of  Akaike  [33]  for  the  noise-free  case,  and  complements 
die  work  of  Pagano  [34]  where  nonlinear  regression  method  is  used.  The  advantage  of  using  the  (linear) 
high-order  Yule-Walker  equations  is  the  simplicity  of  implementation. 


E.  Adaptive  Linear  Estimation  Algoriths  for  Dependent  Data 

In  recent  years  adaptive  linear  estimation  based  on  the  principle  of  steepest  descent  and  its 
variations  have  been  applied  in  a  wide  range  of  problems  such  as  filtering  [35],  pattern  recognition  [36], 
line  enhancement  [37],  antenna  processing  [38],  and  interference  suppression  in  spread-spectrum 
communication  systems  [39].  Probably  the  most  widely-used  algorithm  is  the  so  called  "Widrow  LMS" 
algorithm,  which  uses  the  gradient  method  to  find  the  direction  of  the  steepest  descent  and  at  each 
iteration  replaces  the  true  gradient  vector  by  its  instantaneous  estimate.  The  main  advantage  of  the  LMS 
algorithm  is  the  simplicity  and  the  relative  low  complexity  of  its  implementation.  The  algorithm  is 
defined  as  follows.  Suppose  [ajj  and  \xj)  are  two  jointly  stationary  second-order  processes  representing 
the  desired  signal  and  observation  process,  respectively.  Suppose  we  wish  to  estimate  ak  from  n 
observations  xj  ,  j=k-n+l,...,k  in  a  linear  fashion 

n 

ak  =  £  ^  xk.l+]  ~  hT  xk 

;=i 

where 

h  =  (hl,...,hn)T 
and 

xk  =  (xk-n+x . xk)T  .  (3) 

The  optimal  linear  mean-square  filter  h*  which  minimizes  the  error  £  [(a*  -  ak )2]  is  given  by 

h*  =  R-x  b 


where  R_  is  the  covariance  matrix  of  the  data  vector**, 

£  =  £[***[] 

and  b  is  given  by 


b  =  E[akxk\ . 


When  the  second-order  moments  of  [aj]  and  [*;]  are  unknown,  and  b  are  not  available  to  determine 
the  optima]  filter  h*.  The  LMS  algorithm  adapts  the  filter’s  coefficients  to  the  incoming  data:  at  the 
(&+1)-1'  "instant",  we  have  an  estimate  hk+\  of  h *  given  recursively  by 

hk+ 1  +  (a*  -xTk  hk),k= 1,2,... 


(4) 
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hi  =0 

so  that  the  value  of  h  is  updated  with  the  Lth  incoming  block  of  data  x*.  The  constant  (a  is  the  adaptation 
step  size  of  the  algorithm.  The  signal  estimator  at  the  k‘h  instant  is  then 

ak  =  hk  xk  . 

The  LMS  algorithm  (4)  is  simple  to  implement  and  requires  little  storage.  Furthermore  it  is  applicable 
not  only  to  stationary  inputs  but  also  to  slowly  time-varying  processes  by  tracking  system  parameters  in  a 
nearly  optimum  way.  In  terms  of  the  convergence  properties  of  the  LMS  algorithm,  one  is  usually 
concerned  with  establishing  asymptotic  expressions  for  the  filter’s  coefficients  estimation  error 

Eli  h-h*  II2 

and  for  the  signal  estimation  error 

E[(ak-ak)2} 

when  the  number  of  adaptation  steps  k  becomes  large. 

Under  the  assumption  of  independent  input  vectors  [x,],  the  performance  of  LMS  algorithm  has 
been  extensively  studied  in  the  literature  (see  [35],  [40]  and  the  references  therein).  The  assumption  of 
independent  input  vectors  makes  the  analysis  more  tractable  but  in  many  applications  it  is  not  a  realistic 
model.  Furthermore,  for  scalar-valued  processes  [xy],  the  assumption  of  independent  input  vectors  [xj]  - 
see  (3)  -  cannot  hold  since  successive  vectors  share  all  but  one  of  their  components.  Analysis  of  the  LMS 
algorithms  when  the  input  vectors  [x,  ]  arc  correlated  is  considerably  more  complex  due  to  the  highly 
nonlinear  nature  of  the  algorithm.  Some  results  are  available  in  the  literature  under  fairly  restrictive 
conditions:  an  assumption  of  M-dcpendence  on  the  observation  process  [x;]  is  made  in  [41];  an 
assumption  of  bounded  conditional  moments  is  made  in  [42]  which  excludes  the  important  Gaussian  data 
case. 

During  the  the  contract  period,  we  initiated  a  comprehensive  study  of  the  convergence  properties  of 
the  LMS  algorithm  and  its  variants  under  general  dependence  structure  on  the  signal  and  observation 
processes  { Uj }  and  (xy).  In  particular  we  investigated  the  following  aspects. 

i)  In  [43]  we  provided  a  convergence  analysis  of  a  modified  LMS  algorithm  under  the  fairly  weak 
assumption  that  the  input  processes  satisfy  a  mixing  condition  (strong  mixing  or  p  mixing).  In  this 
analysis  it  was  assumed  that  the  optimal  Wiener-Hopf  coefficients  are  known  to  lie  in  a  bounded  subset  of 
Rn\  however,  it  was  shown  that  for  broad  classes  of  random  processes  such  sets  can  be  determined 
without  actually  solving  the  Wiener-Hopf  equations.  The  results  of  this  work  provide  sharper  bounds  on 
the  mean-square  error  of  the  coefficients’  and  signal  estimates  than  those  obtained  in  [44], 

ii)  In  [45]  we  provided  a  convergence  analysis  of  a  constrained  LMS  algorithm  for  dependent 
processes  satisfying  various  mixing  conditions.  The  need  for  such  algorithms  arises  from  the  fact  that  in 
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many  applications  the  optimal  finite  impulse  response  filter  of  order  n  is  restricted  such  that  its 
coefficients  lie  in  a  subset  of  the  n-dimensional  vector  space  Rn.  For  example,  in  antenna  beam-forming 
the  need  to  control  the  magnitude  of  the  sidelobes  imposes  such  a  constraint  [46][47][48].  Similar 
situations  arise  in  parametric  spectral  estimation  such  as  the  Pisarenko  harmonic  retrieval  [49].  In  these 
references,  the  performance  of  the  constrained  LMS  algorithm  was  evaluated  either  by  simulation  or 
under  the  assumption  of  independent  input  vectors.  In  [45]  we  considered  two  distinct  constraint  sets:  A 
bounded  hypercube  in  Rn,  which  leads  to  a  magnitude-constrained  algorithm,  and  a  bounded 
hypersphere,  which  leads  to  a  quadratically-constrained  algorithm.  The  purpose  of  the  algorithm  is  to 
provide  a  linear  estimate  ak  of  ak  from  the  observation  vector  xk  =  . .  .  ,xk  ,)  where  =  **_,+ 1- 

This  estimate  is  of  the  form 

ak 

where  w  is  the  vector  of  filter  coefficients,  w-ith  a  fixed  dimensionality  n,  constrained  to  a  bounded  set  S  in 
Rn.  When  the  second-order  statistics  of  the  processes  [a;]  and  {*,}  are  known,  the  optimal  mean-square 
filter  coefficients  vv.  is  obtained  by  solving  the  following  problem: 

min  E[ak~ak]2 

we  S 

where,  for  a  magnitude-constrained  algorithm,  the  set  S  is  defined  by 

S  =  {vv:  [w-,1  <B,  i=l . n]  , 

w,  is  the  ith  component  of  vv,  B  is  a  finite  constant;  for  a  quadratic-constrained  algorithm,  the  set  S  is 
given  by 

S  =  [vv  :||vv||  <B]  , 

where  ||  ||  is  the  Euclidean  norm  in  Rn.  We  show  in  [45]  that  the  adaptive  algorithm,  when  the  second 
order  statistics  of  { and  {*;}7=-»  ^  unknown,  takes  die  fonn 

=P[(I-xkxl)  wk  +nakxk],  k= 0,1 . 

a<=xkwk,  k= 0,1 . 

where  the  operator  P  is  a  projection  from  Rn  to  S.  We  show  in  [45]  that  for  input  processes  obeying 
fairly  weak  mixing  conditions,  we  have 

lim  sup  E!|vv*  <;x  C\  (5) 

k—**>  — 


and 


E[ak  -  ak]2  =  e.  +  e* 


with 
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lim  sup  e*  <p  Ci  .  (6) 

k-** o 

Here  e.  is  the  minii  rum  MSE  corresponding  to  the  optimal  constrained  Wiener-Hopf  filter  w.,  e*  is  the 
excess  MSE  of  the  algorithm,  and  C,  i  =  1,2  are  positive  constants.  Note  that  the  bounds  in  (5)(6)  are 
proportional  to  the  adaptation  step  size  fi  and  thus  have  the  same  form  as  the  bounds  established  in  the 
literature  for  the  unconstrained  LMS  algorithm  with  dependent  data  (e.g.  |4 1]  [42]). 


F.  Inference  for  Continuous-Time  Processes  from  Sampled  Data 

Let  X  =  {X (r),  -~=<  t  <°°)  be  a  continuous-time  stationary  process,  [r*]  be  the  sampling  instants, 
and  {X(fi)}  be  the  discrete-time  observation  processes.  We  are  interested  in  estimating  the  statistical 
structure  of  the  process  (X(r),  -<»<  f  <<*=}  from  a  finite  set  of  discrete-time  observations  [X(r*)]*=i. 
Particular  functions  of  interest  are  the  family  of  finite-dimensional  distributions  and  densities  of  the 
process  X,  the  correlation  tunction  C(t)  and  spectral  density  0(A)  of  the  process  X.  Clearly  if  the  the 
sampling  instants  [r* }  are  equally-spaced,  consistent  estimates  of  the  joint  densities  of  the  process  X  from 
the  observations  [X(q.)}  is  not  feasible;  similarly,  if  the  process  X  is  not  bandlimited,  consistent  estimates 
of  C  (t)  and  0(A)  from  equally-  spaced  observations  is  not  possible  due  to  aliasing. 

During  the  contract  period  we  considered  in  [50]  the  problem  of  consistently  estimating  the  famhy 
of  finite-dimensional  densities  of  the  process  X  from  discrete-time  observations.  Let 
f(x;r)=f(xI,...,xm+i;Ti,T2,.~,Tm )  be  the  joint  probability  density  of  the  random  variables 
X(0),X(T1).A'(T2),...,X(Tm),  0  <  Ti  <t2  <...<  Tm  ,  which  is  assumed  to  exist.  Let  K(x)  be  a  bounded 
nonnegative  function  on  /?m+I  satisfying 

f  K{x)dx  =  1  ,  lim  ||x||m+1  K(x)  =  0  , 

and  let  Kn(x)  =  (\/b™*l)K(x/bn).  Similarly,  let  W(u)  be  a  bounded  nonnegative  function  on  Rm 
satisfying 

\W(u)du  =  \,  lim  [|uir  W(u)  =  0, 

R-  ll“ll-*“ 

and  put  \Vn(u)-(\lb^)W{ulbn).  Given  the  observations  [X(r;),r;]"=f  we  estimate  f{rXj  by 

fn(x\T)  =  l/(na(T))  £  Wn(x-Dj)Kn(x-Xj) 

-  j=\ 


where 


m~l 

a(T)=p( Ti)  n  p  (Ti+1— T;), 
—  1=1 


X)  =  (X(r;).X(ry+,) . X(tj+m))  and  Dj  . »>«,-»>).  Here  the  sampling  instants  [r,}n=i 

arc  assumed  to  be  a  renewal  process  on  [0,«=);  p(t)  is  the  probability  density  function  of  the  interarrival 
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limes  which  is  assumed  to  be  positive  on  [0,«)  with  finite  mean  — .  We  allow  the  average  sampling  rate 

p  to  take  any  positive  value  and  consequently,  no  minimum  sampling  rate  /3  is  required.  We  established 
in  [50]  the  quadratic-mean  convergence  of/B(x;r)as  the  sample  size  n  tends  to  infinity  along  with  precise 
asymptotic  expressions  for  the  bias  and  variance/covariance.  In  addition,  we  established  the  asymptotic 
normality  of  the  estimators  These  results  in  [50]  are  shown  to  hold  for  continuous-time  processes 

X  which  are  either  strong  mixing  orp  mixing. 

During  the  contract  period  we  were  also  concerned  with  sampling  design  problems  of  continuous¬ 
time  processes  [51]  [52]:  Let  {X(t),  -<»</<«>}  be  a  second-order  continuous-time  process  with  zero 
mean  and  correlation  function  R(t,s).  Many  problems  in  detection  and  estimation  involve  the  evaluation 
of  the  integral 

T 

I  =  jf(t)X(t)d: 
o 

where  /  is  a  given  function.  In  practice,  the  evaluation  of  /  is  carried  out  in  a  digital  fashion;  the 
observation  process  X  is  sampled  at  n  instants  Tn  ={*„.  lt  •  •  •  ,t„  „}  satisfying 

0 <  t„,  t  <f„, 2  <  <tn,n<T, 

and  based  on  the  samples  [X  (r„  , ) }  ?=1 ,  an  approximation  of  the  integral  /  of  the  fonm 

/«=  Z  Cn.iX0n,i) 
i'=l 

is  used.  One  is  then  concerned  with  the  performance  of  such  approximation  measured  in  terms  of  the 
mean-square  error  e\  =  E{1  -  l„]2.  Clearly  this  depends  on  the  choice  of  the  coefficients  {c'n  t)1=  i  which 
may  be  optimal  or  suboptimal,  and  on  the  choice  of  the  sampling  instants  Tn.  The  sampling  design 
problem  is  to  find  the  best  location  of  the  sampling  points  for  fixed  sample  size  n  or,  asymptotically,  as 
the  sample  size  tends  to  infinity.  For  a  given  choice  of  the  coefficients  and  the  sampling  points,  optimal 
or  not,  one  also  desires  to  find  the  rate  of  convergence  to  zero  of  the  error  as  n  —>  <*>  which  can  be  used 
to  determine  the  required  sample  size  n  for  an  acceptable  level  of  accuracy  of  the  discrete-time 
approximation.  We  remark  that  in  the  context  of  regression  problems,  sampling  designs  were  considered 
earlier  by  Sacks  and  Ylvisaker  [53]-[55].  Sampling  designs  for  suboptimal  integral  approximations  were 
considered  by  Schoenfelder  [56]  for  the  case  of  processes  with  no  quadratic-mean  derivatives. 

In  [51]  the  asymptotic  performance  of  linear  predictors  of  continuous-time  stationary  processes 
from  observations  at  n  sampling  instants  on  a  fixed  interval  was  studied.  Both  optimal  and  simpler 
choices  of  predictor  coefficients  were  considered  using  uniform  and  nonuniform  sampling  schemes.  The 
focus  was  on  processes  with  rational  spectral  density.  When  the  process  has  no  quadratic-mean 
derivatives,  it  was  shown  that  both  the  optimal  and  suboptimal  predictors  have  the  same  rate  of 
quadratic-mean  convergence  n~ 2  and  the  same  asymptotic  constant  which  depends  on  the  sampling 
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design.  The  performance  of  the  asymptotically  optimal  design  is  compared  analytically  and 
computationally  to  that  of  uniform  sampling  and  the  improvement  factor  is  determined.  When  the  process 
has  exactly  one  quadratic-mean  derivative,  and  a  suboptimal  coefficients  are  used,  it  is  shown  that 
uniform  sampling  has  a  rate  of  quadratic-mean  convergence  of  only  n~x  while  nonuniform  sampling  can 
be  designed  to  achieve  a  rate  of  n-4. 

In  [52]  we  considered  the  problem  of  Monte  Carlo  approximation  of  the  integral  of  weighted 
random  processes.  The  simplest  Monte  Carlo  method  for  approximating  the  integral 

l 

HfX)  =  \f{t)X(t)dt 
o 

of  a  function  /  over  a  finite  interval,  uses  n  independent  samples  U  . . Un  from  a  uniform  distribution 

over  the  unit  interval,  and  forms  the  average  estimate 

j*m  =  -1  £  rmxm. 
n  «=! 

The  mean-square  error  of  Jn(f)  is 

1  1  11 

E[I(fX)-Jn(fX)]2  =  -  {\f2(x)R(x,x)dx-\\f(x)R(x,y)f(y)dxdy} 
n  0  00 

and  thus  tends  to  0  at  the  rate  of  ft'1.  No  improvement  in  the  rate  of  convergence  is  generally  expected 
from  any  additional  smoothness  assumptions  on /or  on  X. 

Motivated  by  the  work  of  Yakowitz,  Krimmel  and  Szidarovszky  [57]  for  the  integral  of  nonrandom 
functions,  wc  considered  in  [52]  the  convergence  properties  of  a  trapezoidal  Monte  Carlo  approximation 
based  on  the  ordered  sample  tn,o  =  0  <  f„ti  <  tBi2  <  •  ■  ■<  h,,„  <  1  =  tn,n+i  obtained  from  the 
independent,  uniformly  distributed  samples  f/j,  U 2  ,...,  Un.  The  approximation  of  I  (fX)  is  now  given  by 

In  m  =  \  £  {f  (rn  i)  X{tnJ)  +  f(tntM)X(tn>i+ 1)]  (t„,l+,  ~tn,i)  . 

1  1=0 

We  show  in  [52]  that  when  /has  two  continuous  derivatives  on  [0,  1]  and  the  processes  X  has  one 
quadratic-mean  derivative,  the  rate  of  mean-square  convergence  of  the  approximation  ln  ifX)  is  precisely 
n-4,  i.e., 

lim  n 4  E[I(fX)-In(fX)]2  =  C(f,R ) 


where  the  asymptotic  constant  C  (/,/?),  which  depends  on  /  and  on  correlation  function  R ,  is  determined 
explicitly. 
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